--- title: Sensors keywords: fastai sidebar: home_sidebar summary: "Reads and packages IMU/GPS/Pressure-Humidity-Temperature sensor data over UART. Used for georectification. " description: "Reads and packages IMU/GPS/Pressure-Humidity-Temperature sensor data over UART. Used for georectification. " nb_path: "nbs/07_sensors.ipynb" ---
{% raw %} 
No module named RPi or Jetson. Did you try `pip install RPi.GPIO` or `pip install Jetson.GPIO`?
{% endraw %}

{% include tip.html content='This module can be imported using from openhsi.sensors import *' %}{% include warning.html content='Still experimental. Stay tuned. ' %} The OpenHSI camera requires motion to generate 2D spatial datacubes. Yet, motion also introduces other artefacts that need georectification. To correct for spurious motion, we need to collect absolute orientation and geolocation of the camera simultaneously with the camera capture. This is where this module comes in. An IMU, GPS, and Pressure/Humidity/Temperature sensor needs to be read and recorded.

A Teensy 4.0 operates all the sensors, and devices using a Real Time Operating System (RTOS) like cooperative scheduler to run each component update at the desired frequency. By reducing the I/O, and CPU load on the main development board (the Raspberry Pi 4 with 8 GB RAM), the sensor updates are offloaded to a microcontroller with a real time clock to sync and timestamp each sensor measurement. The whole thing is assembled onto a PCB that stacks with the Raspberry Pi 4 and battery hat.

Component Rate (Hz) Info
Teensy 4.0 (uC) 24 MHz Clock ---
NEO 9N (GPS) 20 Hz I2C @400 kHz (takes ~20 ms per update)
BNO055 (IMU) 100 Hz I2C_1 @400 kHz
BME280 (Air) 100 Hz I2C_1 @400 kHz
DS3231 (RTC) 100 Hz I2C_1 @400 kHz
XBee 1 Hz UART_1 @115,200 Hz (~2 ms per update)
Raspberry Pi 4 packets @100 Hz UART @921,600 Hz (~0.8 ms per update)
Start button 4 Hz poll button linked to LED notifying status

An XBee is also programmed to check sensor status remotely during operation. This could be useful to diagnose any issues without being physically connected to the microcontroller. A basic streaming dashboard is included.

{% raw %}
{% endraw %} {% raw %}
{% endraw %}

Each data packet contains timestamped sensor data. The item fields are then extracted from the raw binary serial stream. {% include note.html content='The Teensy runs a 32 bit Cortex-M7 so serial packets are padded. In other words, the data struct is padded in contiguous memory so a byte variable followed by float variable will include 3 unused bytes in-between so things are packed as 32 bits at a time. ' %}

{% raw %}
{% endraw %}

The data packet is sent as a C struct so we need to decode the binary stream and interpret each byte as the corresponding C type. In each packet, there are status bytes to indicate which sensor has been updated.

{% raw %}

decode_packet[source]

decode_packet(buff:byte string=None)

Decode `buff` into a list of decoded variables
{% endraw %} {% raw %}
{% endraw %}

It's fairly safe to assume that all data saving will occur on a separate storage device. I tested this with an SSD mounted over USB.

{% raw %}

class SensorStream[source]

SensorStream(baudrate:int=921600, port:str='/dev/serial0', start_pin:int=17, ssd_dir:str='../../../media/pi/fastssd/', cam_name:str=None)

Parses ancillary sensor data for saving
{% endraw %} {% raw %}
{% endraw %} {% raw %}

SensorStream.read_packet[source]

SensorStream.read_packet(header:chr=`b''*, **num_bytes**:int=*76*, **timeout**:float=*2.0`*)

Reads at least `num_bytes` of a data packet starting with `header`
and times out after `timeout` seconds if packet is invalid.

SensorStream.save[source]

SensorStream.save()

Save the data packets. Will save some plots of the data as well.

SensorStream.master_loop[source]

SensorStream.master_loop(n_lines:int=128, processing_lvl:int=0, json_path:str=None, pkl_path:str=None, preconfig_meta:str=None, ssd_dir:str=None, switch_pin:int=17)

Continuous run saving packets during start button pressed. If you want to capture camera as well,
input all the optional parameters.
Type Default Details
n_lines int 128 how many along-track pixels
processing_lvl int 0 desired processing done in real time
json_path str `` path to settings file
pkl_path str `` path to calibration file
preconfig_meta str `` path to metadata file
ssd_dir str `` path to SSD
switch_pin int 17 button that controls collection

SensorStream.clean_df[source]

SensorStream.clean_df(df:DataFrame)

Converts time offsets in `df` into datetime and splits sensor readings that update
at different rates. Also saves the plots as a picture.
{% endraw %}

{% include tip.html content='The serial port for Raspberry Pi is "/dev/serial0". For the Jetson, it is "/dev/ttyTHS0".' %}{% include note.html content='GPS PPS callbacks are experimental. I haven’t found a way to use them effectively.' %}

{% raw %}

set_pps_cb[source]

set_pps_cb(gps_pin:int=19, times_list:List[datetime]=[], bouncetime_ms:float=10)

Setup a callback that appends the system time to `times_list` each time
a GPS pulse per second is detected on `gps_pin`.
Type Default Details
gps_pin int 19 GPS pulse per second pin
times_list datetime] `` Any list to append system time when callback is called
bouncetime_ms float 10 Debouncing time for the GPS PPS signal
{% endraw %} {% raw %}

clear_pps_cb[source]

clear_pps_cb(gps_pin:int=19)

Clear the GPS pulse per second callback on `gps_pin`.
Type Default Details
gps_pin int 19 GPS pulse per second pin
{% endraw %} {% raw %}
{% endraw %}

Let's now test this using simulated ancillary sensor data packets.

{% raw %}

collect_sim[source]

collect_sim(rtc_offset_ms:float=0)

Generate fake sensor packets for testing.
{% endraw %} {% raw %}
{% endraw %} {% raw %} 
ss = SensorStream(baudrate = 921_600,
                  port = '/dev/serial0',
                  start_pin = 17,
                  ssd_dir = '.')

ss.packets = []
for i in tqdm(range(77)):
    ss.packets.append(collect_sim(rtc_offset_ms=150))
    time.sleep(0.01)

#ss.save()

100%|███████████████████████████████████████████| 77/77 [00:00<00:00, 90.49it/s]
{% endraw %} {% raw %} 
#hide_output

ss = SensorStream(baudrate = 921_600,
                  port = '/dev/serial0',
                  start_pin = 17,
                  ssd_dir = '/media/pi/fastssd')

ss.master_loop()
{% endraw %} {% raw %} 
#hide_output

ss = SensorStream(baudrate = 921_600,
                  port = '/dev/serial0',
                  start_pin = 17,
                  ssd_dir = '/media/pi/fastssd',
                  cam_name="FlirCamera")

ss.master_loop(n_lines=256,
               processing_lvl=2,
               json_path="/media/pi/fastssd/cals/OpenHSI-FLIR01/OpenHSI-FLIR01_settings_Mono8_bin1.json",
               pkl_path="/media/pi/fastssd/cals/OpenHSI-FLIR01/OpenHSI-FLIR01_calibration_Mono8_bin1.pkl",
               preconfig_meta=None,
               ssd_dir="/media/pi/fastssd",
               switch_pin=17)
{% endraw %} {% raw %}

interp2camera_times[source]

interp2camera_times(df:DataFrame, ts:array, save:bool=False)

Interpolate the ancillary sensor data to the timestamps for when
each frame was taken with the camera.
{% endraw %} {% raw %}
{% endraw %}

Streaming dashboard

View the XBee status on a dashboard that shows the last 100 points. This is a simple implementation (it is possible to improve the front end using something like Dash and plotly).

{% raw %}
{% endraw %} {% raw %}

class SensorDashboard[source]

SensorDashboard(baudrate=115200, port='/dev/cu.usbserial-DN05TVTD', buff_len:int=100)

A dashboard for viewing ancillary sensor status.
{% endraw %} {% raw %}
{% endraw %} {% raw %}

SensorDashboard.run[source]

SensorDashboard.run()

Infinite loop to check sensor status every second.

SensorDashboard.__call__[source]

SensorDashboard.__call__()

Create layout of panels

SensorDashboard.clear_all[source]

SensorDashboard.clear_all(event)

Clears all the sensor streams

SensorDashboard.update[source]

SensorDashboard.update()

Push new sensor data to streams

SensorDashboard.read[source]

SensorDashboard.read(timeout:float=2)

Parse XBee data packets and timeout if none received.
{% endraw %} {% raw %} 
#hide_output

sd = SensorDashboard()
sd()
{% endraw %}

{% include warning.html content='Start/Pause button not working!' %}

{% raw %} 
#hide_output

sd.run()
{% endraw %}

The SensorDashboard will save the data coming in which can be accessed in a pd.DataFrame. Here is some experimental data with noise added to the latitude/longitude points so the ESRI map loads.

{% raw %} 
sd.data_df.head(10)
lat lon sats temp pressure humidity sys_cal gyro_cal accel_cal mag_cal
0 -33.743084 152.002595 6 23.959999 1009.301147 57.174805 0 0 0 0
1 -33.628404 151.289836 6 23.969999 1009.341736 57.151367 0 0 0 0
2 -32.957323 151.410112 6 23.969999 1009.320557 57.151367 0 0 0 0
3 -32.974990 151.678872 6 23.990000 1009.344849 57.945312 0 0 0 0
4 -33.148673 151.656227 6 23.990000 1009.343933 58.179688 0 0 0 0
... ... ... ... ... ... ... ... ... ... ...
603 -32.979386 151.482988 9 24.820000 1009.469055 56.936523 3 3 3 3
604 -33.500686 151.774686 9 24.830000 1009.441711 56.960938 3 3 3 3
605 -33.438636 151.022355 9 24.820000 1009.461975 56.960938 3 3 3 3
606 -32.848062 151.060729 9 24.820000 1009.477783 56.998047 3 3 3 3
607 -33.395096 151.709758 9 24.809999 1009.464233 57.022461 3 3 3 3

608 rows × 10 columns

{% endraw %}